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Abstract 

Virtual  Reality  (VR)  has  beeome  mature  enough  to  be  sueeessfully  used  in  elinieal  applieations  sueh  as  exposure 
therapy,  pain  distraetion  and  neuropsyehologieal  assessment.  Flowever,  we  now  need  to  go  beyond  the  outeome  data 
from  this  researeh  and  eonduet  the  detailed  seientifie  investigations  required  to  better  understand  what  faetors 
influenee  why  VR  works  (or  doesn’t)  in  these  types  of  elinieal  applieations.  This  knowledge  is  required  to  guide  the 
development  of  VR  applieations  in  the  key  areas  of  edueation,  training  and  rehabilitation  and  to  further  evolve 
existing  VR  approaehes.  One  of  the  primary  assets  obtained  with  the  use  of  VR  is  the  ability  to  simulate  the 
eomplexity  of  real  world  environments,  within  whieh  human  performanee  ean  be  tested  and  trained.  But  this  asset 
eomes  with  a  priee  in  terms  of  the  eapture,  quantifieation  and  analysis  of  large,  multivariate  and  eoneurrent  data 
sourees  that  refleet  the  naturalistie  behavioral  interaetion  that  is  afforded  in  a  virtual  world.  As  well,  while  aehieving 
realism  has  been  a  main  goal  in  making  eonvineing  VR  environments,  just  what  eonstitutes  realism  and  how  mueh  is 
needed  is  still  an  open  question  situated  firmly  in  the  researeh  domain.  Just  as  in  real  “reality”,  sueh  faetors  in  virtual 
reality  are  eomplex  and  multivariate,  and  the  understanding  of  this  eomplexity  presents  exeeptional  ehallenges  to  the 
VR  researeher.  For  eertain  researeh  questions,  good  behavioral  seienee  often  requires  eonsistent  delivery  of  stimuli 
within  tightly  eontrolled  lab-based  experimental  eonditions.  However,  for  other  important  researeh  questions  we  do 
not  want  to  eonstrain  naturalistie  behavior  and  limit  VR’s  ability  to  replieate  real  world  eonditions,  simply  beeause  it 
is  easier  to  study  human  performanee  with  traditional  lab-based  methodologies.  By  doing  so  we  may  eompromise 
the  very  qualities  that  eomprise  VR’s  unique  eapaeity  to  mimie  the  experienees  and  ehallenges  that  exist  in  everyday 
life.  What  is  really  needed  to  address  seientifie  questions  that  require  natural  exploration  of  a  simulated 
environment  are  more  usable  and  robust  tools  to  instrument,  organize,  and  visualize  the  eomplex  data  generated  by 
measurements  of  partieipant  behaviors  within  a  virtual  world.  This  paper  will  briefly  deseribe  the  rationale  and 
methodology  of  an  initial  study  in  an  ongoing  researeh  program  that  aims  to  investigate  human  performanee  within  a 
virtual  environment  where  uneonsfrained  “free  will”  exploratory  behavior  is  essential  to  researeh  questions  that 
involve  the  relationships  between  physiology,  emotion  and  memory.  After  a  diseussion  of  the  researeh  protoeol  and 
the  types  of  data  that  were  eolleeted,  we  deseribe  a  novel  tool  that  was  home  from  our  need  to  more  effieiently 
eapture,  manage  and  explore  the  eomplex  data  that  was  generated  in  this  researeh.  An  example  of  a  researeh 
partieipant’s  annotated  display  from  this  data  management  and  visualization  tool  is  then  presented.  It  is  our  view 
that  this  tool  provides  the  eapaeity  to  better  visualize  and  understand  the  eomplex  data  relationships  that  may  arise  in 
VR  researeh  that  investigates  naturalistie  free  will  behavior  and  its  impaet  on  other  human  performanee  variables. 


1  The  Challenge  for  Virtual  Reality  Research 

Virtual  reality  (VR)  has  been  transforming  the  domain  of  elinieal  therapy  by  providing  a  plausible  simulaerum  to 
real  world  experienees.  Perhaps  the  applieation  of  VR  for  the  treatment  of  anxiety  disorders,  espeeially  simple 
phobias,  provides  the  most  eonsistent  reports  of  positive  therapeutie  outeomes  (see  reviews  in  Glantz,  Rizzo  & 
Graap,  2003;  Rizzo  &  Kim,  2005).  This  approaeh  has  allowed  for  patients  to  be  immersed  within  virtual 
environments  that  provide  the  graduated  therapeutie  exposure  to  relevant  stimuli  that  are  the  foeus  of  their  fear  or 
eondition.  While  it  is  now  elear  that  these  therapeutie  effeets  exist,  there  is  mueh  to  be  done  to  understand  the 
seientifie  basis  for  sueh  VR  effieaey.  In  order  to  examine  virtual  reality  beyond  the  realm  of  elinieal  outeomes,  new 
kinds  of  tools  are  required  that  faeilitate  the  seientifie  study  of  the  stimulus  parameters  and  resulting  responses  that 
oeeur  in  sueh  simulated  worlds. 

Virtual  reality  is  hampered  from  full  experimental  validity  by  the  very  attribute  that  makes  it  an  exeellent 
deviee  for  the  replieation  of  real  world  ehallenges:  it  is  exeeptionally  eomplex  and  multivariate.  A  typieal  VR 
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session  generates  immense  amounts  of  data,  even  for  a  relatively  short  experience  (8-10  minutes),  and  VR  worlds 
that  allow  for  each  subject  to  make  their  own  choices  within  the  environment  further  complicate  this  data  collection 
and  analysis.  This  type  of  “free-will”  experience  can  be  subverted  by  constraining  the  subject’s  path  or  choices 
within  the  environment  (Morie  2002),  but  some  research  questions  are  better  served  by  permitting  such  agency, 
especially  those  in  which  observing  the  participant’s  uncoerced  choices  are  the  focus  of  study.  However,  in-depth 
investigation  of  agency  in  VR  is  not  a  simple  task  and  as  yet  no  specialized  tools  exist  to  instrument,  organize,  and 
visualize  the  ensuing  complex  data  produced  in  such  free  will  scenarios.  This  paper  will  present  a  research  protocol 
that  allows  such  agency  in  a  VR  application  to  investigate  naturalistic  free  will  behavior  and  its  impact  on  other 
human  performance  variables  (i.e.,  physiology,  emotion  and  memory).  This  will  involve  a  discussion  of  the  research 
question,  the  types  of  data  that  was  collected,  and  will  describe  a  novel  tool  that  was  home  from  our  need  to  more 
efficiently  capture,  manage  and  explore  the  complex  data  that  was  generated  in  this  research.  An  example  of  an 
annotated  display  from  a  research  participant  will  also  be  presented.  We  believe  this  tool  will  provide  the  capacity  to 
better  visualize  and  understand  the  complex  data  relationships  that  may  arise  generally  in  VR  research  and  lead  to  a 
more  thorough  understanding  of  the  unique  assets  and  limitations  that  VR  offers  human  behavioral  research. 

2  The  Sensory  Environments  Evaluation  Research  Program 

At  the  Institute  for  Creative  Technologies  (ICT),  the  Sensory  Environment  Evaluation  (SEE)  research  program  is 
building  and  investigating  a  series  of  virtual  environments  (VEs)  focusing  on  arousal  states  and  behaviors  when 
humans  naturalistically  interact  within  emotionally  evocative  scenario  conditions.  Our  first  experiment  evaluates  the 
memory  durability  afforded  by  an  emotionally  rich  VE,  with  the  ultimate  goal  of  making  more  effective  virtual 
worlds  for  learning.  In  addition,  we  are  studying  the  effects  of  priming  on  both  behavior  and  the  emotional  valence 
of  the  VE  experience.  This  research  was  inspired  in  part  by  recent  findings  that  clearly  show  interconnectedness 
among  emotions,  cognition,  learning  and  behavior.  This  research  also  indicates  that  emotions  cannot  be  easily 
dismissed  when  focusing  on  pedagogical  goals  (cf  Damasio,  1994;  Dolan,  2002;  Role,  1999).  Emotions  can  also 
heighten  the  attention  and  response  to  visual  cues  (Lane,  Chua  &  Dolan,  1999),  and  this  may  be  true  for  other  senses 
as  well. 

Ultimately  we  would  like  to  empirically  measure  emotional  responses  of  the  VE  participants  under  varying 
conditions  that  systematically  manipulate  the  challenges  that  typically  occur  during  natural  exploration  in  the  real 
world.  Within  this  research,  we  use  standard  psychophysiological  measures  of  heart  rate  (EKG)  and  skin 
conductance  (SCR)  to  capture  arousal  states.  While  these  measures  cannot  tell  us  what  specific  emotions  are  being 
experienced,  they  do  provide  information  on  participants’  reactions  that  can  be  correlated  in  time  to  the  various 
emotional  trigger  stimuli  that  we  deliver  within  the  virtual  world.  We  also  record  detailed  behavioral  data  of  each 
participant’s  exploratory  actions  within  the  virtual  world,  typically  encompassing  a  five  to  ten  minute  session 
sampled  at  20  frames  per  second  per  subject.  As  well,  we  are  investigating  the  relationship  between  subject 
characteristics  (i.e.,  immersive  tendencies,  computer  familiarity,  time  spent  playing  computer  games,  etc.), 
physiology,  naturalistic  exploration,  and  how  this  influences  memory  for  the  events  that  occur  in  the  virtual  world. 

As  this  research  progressed,  we  quickly  realized  that  the  amount  of  data  we  were  collecting  to  support  our 
investigation  was  vast  and  unwieldy.  After  the  first  round  of  65  test  subjects  in  our  initial  study,  we  found  it 
necessary  to  temporarily  move  our  focus  to  creating  tools  that  would  allow  us  to  adequately  examine  the  data  our 
study  was  generating.  We  call  this  evolving  tool  set.  Phloem,  after  the  channels  in  botanical  life  that  serve  as 
nutrient  (information)  support,  transport  and  storage  within  the  plant.  Even  in  its  early  stages.  Phloem  has 
demonstrated  value  for  increasing  our  understanding  of  the  complex  data  produced  in  this  study. 

3  Experiment  Design  Process 

The  SEE  Project’s  investigation  of  emotionally  compelling  VEs  comprises  a  threefold  experimental  approach.  First 
we  base  the  design  process  on  the  various  sensory  inputs  available  to  the  VR  practitioner  (sight,  sound,  smell, 
haptics),  and  attempt  to  determine  how  these  can  be  effectively  combined  to  elicit  emotional  responses.  Second,  we 
test  research  subjects  in  an  emotionally  “provocative”  VE  and  measure  their  physiological  arousal  states  and 
behaviors.  Third,  with  our  findings  from  these  subject  evaluations,  we  expect  to  devise  a  methodology  that  uses 
emotions  to  increase  the  effectiveness  of  virtual  environments  for  a  variety  of  purposes.  Our  first  study  is  examining 
the  influence  of  priming  on  engagement  and  performance  in  a  virtual  world. 

3.1.1  Independent  Variable  -  Priming 

For  our  first  experiment  we  focused  on  one  means  to  establish  an  emotional  connection  between  the  subject  and  the 
VE.  We  chose  the  independent  variable  of  priming,  as  described  in  (Schacter  and  Buckner,  1998)  and  (Tulving  and 


Schacter,  1990),  to  give  participants  a  context  for  what  they  were  about  to  experience.  Such  priming  is  sometimes 
also  referred  to  as  the  instructional  set  and  helps  subjects  construct  a  schema  by  which  their  task  is  understood. 

Each  participant  was  randomly  assigned  to  one  of  two  priming  groups.  In  the  first  group,  participants 
received  priming  that  instructed  them  about  the  goal  of  their  impending  task  in  the  VE  in  the  context  of  an 
entertaining  video  game  that  they  were  about  to  play.  The  second  group  received  a  briefing  that  indicated  they 
would  experience  a  serious  military  training  exercise.  The  priming  process  actually  started  when  the  participants 
were  brought  into  a  room  to  fill  out  demographic  and  self-report  questionnaires  (described  below).  Themed  posters 
decorated  the  walls  of  the  room:  game-based  for  the  game  primed  participants  and  serious  military  posters  for  those 
undergoing  the  second  form  of  priming.  These  were  not  referred  to,  but  rather  formed  a  backdrop  in  the  room.  In 
fact,  if  asked  about  them,  the  evaluators  would  say  they  could  not  comment  about  anything  in  the  room.  The 
principal  priming  mechanism  was  a  two  minute  video  clip  delivered  by  an  appropriately  themed  narrator 
immediately  preceeding  the  VE  experience.  The  basic  instructions,  goals  and  VR  environments  were  exactly  the 
same  for  each  group.  Only  the  manner  of  presentation  and  the  context  differed.  We  tested  64  subjects,  ages  18-40, 
drawn  from  a  standard  university  population,  divided  evenly  across  these  two  groups.  We  were  especially  interested 
in  determining  if  the  two  priming  conditions  would  have  a  significant  affect  on  participants’  behavioral  and  arousal 
measures. 

3.1.2  Subject  Variables 

Our  experiment  used  standard  pre-experience  instruments  to  collect  pertinent  subject  characteristics.  These  included 
a  modified  Immersive  Tendencies  Questionnaire,  or  ITQ  (Witmer  &  Singer,  1998),  the  Simulator  Sickness 
Questionnaire  (SSQ)  (Kennedy,  Lane,  Berbaum  &  Lilienthal,  1993)  and  the  Virtual  Environments  Questionnaire 
(VEQ)  (Usoh,  Catena,  Arman  &  Slater,  2000).  We  made  some  small  modifications  to  the  ITQ  pertaining  to 
questions  about  participants’  history  and  experiences  with  various  types  of  video  games.  This  data  allowed  for  the 
comparison  between  game  players  and  non-game  players  to  see  if  this  subject  characteristic  had  any  bearing  on  the 
outcomes  of  primed  behavior  observed  in  the  simulation.  Our  new  questions  allowed  us  to  further  divide  subjects 
into  the  specific  type  of  games  they  played  and/or  preferred:  first  person  shooter,  role  playing,  puzzle,  etc. 

2.1.2  Subject  Arousal 

Subjects  were  next  outfitted  with  physiological  sensors  to  record  EKG  and  SCR.  The  participant  was  then  brought 
into  the  testing  room  where  the  VR  viewing  and  navigation  equipment  was  explained  and  fitted.  To  minimize 
unfamiliarity  with  the  VR  devices  in  use  (an  InterSense  acoustic  and  inertial  tracker,  with  X  and  Y  dimensions 
mapped  onto  a  joy  stick,  and  Z,  pitch,  roll  and  yaw  mapped  onto  a  sensor  mounted  on  a  Kaiser  Pro-View  head 
mounted  display,  or  HMD)  each  participant  was  given  a  practice  session  in  a  VE  “Tutorial  Room”.  This  neutral 
space  allowed  the  subjects  to  experience  and  practice  the  full  range  of  movements  and  observational  skills  required 
for  the  VR  evaluation  scenario.  We  found  this  to  be  a  necessary  step  as,  absent  such  familiarization,  participants 
often  explored  and  interacted  in  the  virtual  world  with  a  very  limited  range  of  movement,  such  as  not  crouching  or 
bending  down,  and  looking  only  straight  ahead  with  “tunnel  vision”. 

2.1.4  Subject  Behaviors 

Once  the  participant  gained  adequate  interaction  familiarity  within  the  tutorial  room  (determined  by  the  evaluator 
and  subject  together)  the  participant  was  requested  to  return  to  the  starting  point  of  the  tutorial  room  without 
instructor  assistance.  This  established  a  baseline  metric  we  refer  to  as  the  “free-walking  baseline”  for  the 
physiological  data.  It  was  inspired  by  the  “vanilla  baseline”  described  by  Jennings  et  ah,  (1992),  who  suggested 
giving  a  simple  cognitive  task  to  help  standardize  the  mental  activity  of  all  subjects  during  the  baseline  period,  since 
not  all  subjects  will  follow  the  instructions  for  a  resting  baseline  (i.e.  to  sit  quietly,  relax,  keep  eye  closed,  etc.).  A 
free  walking  baseline  should  provide  a  more  natural  baseline  measure  as  the  subject,  being  engaged  in  a  simple  task, 
is  less  affected  by  the  anticipation  of  a  (stressful)  task  or  the  experiment  itself  This  free  walking  baseline  also 
allows  us  to  compensate  for  any  impact  the  overhead  of  using  the  VR  equipment  might  induce  (e.g.  anxiety  due  to 
different  proficiency  levels  in  using  a  joystick  from  subject  to  subject,  using  the  head  mounted  display,  being 
tethered  to  cables,  etc.).  Following  this,  the  core  of  the  experiment  began.  Depending  on  the  random  assignment  of 
conditions,  the  participant  was  shown  one  of  the  two  priming  video  clips  in  the  HMD  that  outlined  the  mission  to  be 
undertaken  in  the  VE. 


3.1.5  Free  agency  experience  and  empirical  research 

The  experiment  required  eaeh  subjeet  to  play  the  role  of  a  forward  seout  in  a  military-style  seenario,  and  to 
determine  whether  rebels  or  refugees  inhabit  an  area  of  suspieion.  In  this  regard,  the  mission  (as  deseribed  in  the 
priming  videos)  is  basieally  an  observational  exereise,  but  we  add  a  simple  objective  of  dropping  a  GPS  transmitter 
near  a  specific  building  if  enough  clues  indicate  that  rebels  are  inhabiting  the  target  area.  Because  this  virtual 
mission  takes  place  at  night  we  have  named  it  DarkCon  (see  Figure  1). 


Figure  1:  A  participant  in  the  HMD  and  a  scene  from  the  VE  experiment  world:  DarkCon 

Given  their  mission  briefing  instructions,  participants  knew  what  they  needed  to  do  and  where  they  needed  to  go  in 
the  DarkCon  environment.  They  could  choose  to  follow  the  mission  to  the  best  of  their  ability,  or  they  might  not. 
Because  of  the  free  will  given  to  the  participants,  each  experience  was,  in  essence,  unique.  Not  having  identical 
experiences  within  experimental  conditions  may  appear  to  be  a  problem  when  embarking  on  studies  in  VE,  but  it  is 
this  agency  that  is  precisely  why  we  are  interested  in  such  experiences.  Each  person’s  choices  and  decisions  within 
the  unscripted  world  are  his  or  her  own.  Such  choice  was  not  only  necessary  for  this  particular  study,  but  also  had 
rich  implications  for  individualized  training  or  education  that  includes  cognitive  decision-making.  An  exception  to 
this  free  agency  is  that  we  made  it  virtually  impossible  to  complete  the  mission  without  being  discovered.  There 
were  several  variations  on  this  discovery  ending  (i.e.,  vicious  dog  attacks,  landmines,  being  shot)  and  all  were 
designed  to  elicit  strong  arousal  states.  We  also  included  a  segment  of  the  enviromuent  where  participants’  motions 
were  naturalistically  more  constrained:  a  culvert  in  which  subjects  start  out  from  that  has  only  one  direction  of 
traversal  in  order  to  get  to  an  outside  environment.  As  long  as  participants  traverse  the  full  length  of  the  culvert  (as 
did  99%  of  them)  then  we  have  a  component  of  the  experience  that  is  roughly  the  same  for  each  subject.  In  this  way 
we  have  three  varying  conditions  within  one  experiment:  a  constrained  path  through  the  culvert  (which  still  allows 
for  free  will  in  terms  of  what  they  look  at  and  how  long  they  linger  in  any  part  of  the  space),  an  unconstrained 
outdoor  environment,  and  a  “high  anxiety-producing”  end  of  scenario. 

We  realize  this  is  a  challenging  approach  to  both  data  collection  and  analysis,  and  consequently  needed  to 
apply  some  structure  to  our  quantification  process.  For  example,  we  divided  the  experience  into  three  distinct 
segments,  or  epochs,  that  corresponded  to  the  varying  degrees  of  free  will  agency  described  above.  The  first  epoch 
was  defined  as  the  period  when  the  subject  was  within  the  constrained  culvert  space.  The  second  is  the  time  in  the 
exterior  environment  (very  unconstrained)  and  the  third  is  at  the  end  of  scenario  segment  where  all  subjects  meet  a 
bad  ending  designed  to  provoke  heightened  arousal. 


3.1.6  Dependent  Variables  -  Memory 

The  primary  dependent  variable  in  this  initial  study  was  memory  for  the  eharaeteri sties  of  the  VE.  The  memory 
measure  assessed  pure  reeall  and  queried  the  partieipant  about  his  or  her  aetions  and  observations  while  within  the 
Darkeon  VE,  ineluding  deteetion  of  sueh  items  as  weapons,  vehieles  and  people.  One  assessment  was  administered 
immediately  upon  eompletion  of  the  experienee  {Immediate  Recall)  and  the  second  assessment  was  conducted  by 
telephone  one  week  later  {Delayed  Recall).  We  termed  the  difference  in  these  two  scores  the  Memory  Durability 
score.  Since  agency  allowed  subjects  to  explore  the  Darkeon  environment  in  unique  ways,  the  meaning  of  these 
values  required  an  “experience”  processing  tool  that  could  indicate  what  objects  the  subject  actually  “saw”  during 
free  will  exploration  of  the  environment. 

4  Data  Mining  in  Multivariate  Dataspace 

Within  our  first  experiment  we  collected  several  types  of  data,  both  qualitative  and  quantitative,  some  of  which  have 
been  previously  described  in  this  paper.  Qualitative  data  included  the  aforementioned  participant  reports: 
questionnaires  administered  both  before  and  after  the  VE  experience  and  the  two  memory  assessments.  The 
quantitative  data  included  the  full  experience  biometric  recordings  of  EKG  and  SCR  measures  that  contained 
annotations  of  significant  events.  Also  collected  was  the  participants’  behavior  in  the  VE  (six  degrees  of  freedom 
including  x,  y  location,  z  altitude  for  stoop/stand,  and  angular  head  orientation  (pitch,  roll  and  yaw)  to  determine 
what  was  being  looked  at).  We  used  some  of  this  raw  data  to  derive  other  metrics,  such  as  heart  rate  (HR)  from  the 
EKG. 

4.1  Visualization  of  behavior  and  arousal  data 

The  sheer  quantity  of  such  participant  data  motivated  development  of  the  Phloem  tool  set  to  more  effectively 
manage  multiple  concurrent  streams  of  data.  The  first  task  was  to  create  a  visual  display  of  the  physiological 
(arousal)  and  behavioral  data. 

4.1.1  Basic  data  capture  and  display 

Phloem  works  in  conjunction  with  the  data  captured  by  the  engine  that  drives  the  real-time  scenario.  This  engine 
includes  information  about  every  object  and  every  event  in  the  experience  and  is  capable  of  producing  a  log  file  that 
contains  this  information.  The  log  file  also  contains  the  numerical  values  of  the  x,  y,  and  z  position  and  head 
orientation  of  the  subject  sampled  20  times  per  second  by  an  InterSense  head  tracking  unit  for  the  duration  of  the 
experience  (typically  5-10  minutes  per  subject).  Phloem  presents  the  data  in  an  easy-to-scan  user  interface,  making 
the  examination  of  the  recorded  data  easier.  Phloem  also  uses  this  data  to  drive  a  perfect  re-enactment  of  the 
experience  by  feeding  it  back  to  control  the  display  parameters  of  the  scene,  allowing  for  a  visual  playback  of  each 
participant’s  actions.  This  playback  is  scrubbable,  like  a  video  recording/playback  tool,  so  a  researcher  can  follow 
participants  forwards  or  backwards,  or  examine  what  they  do  in  detail.  Phloem  automatically  reads  any  key  events 
or  interest  areas  of  any  experience  from  the  engine  data  (for  example,  stimuli  such  as  bats,  or  a  passing  truck  in 
DarkCon)  and  automatically  presents  these  as  choices  to  the  researcher  via  drop  down  menus.  If  an  event  is  chosen 
from  this  menu,  the  playback  will  jump  directly  to  that  section.  Phloem  can  also  read  in  and  display  both  sets  of 
physiological  signals  for  each  subject,  and  show  them  synched  to  the  re-enactment  (see  Figure  2). 


Phloem  screens  of  a  user  experience 
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Figure  2:  Data  as  shown  by  the  most  recent  Phloem  visualization  tool 


4.1.2  More  detailed  queries 

Phloem  goes  beyond  the  playback  features  described  above.  As  it  stores  all  the  data  from  the  experience,  customized 
queries  can  be  designed  to  pull  specific  information  of  interest  from  that  data.  The  behavioral  data  contains 
information  such  as  how  long  a  participant  took  to  go  through  the  world  as  well  as  where  and  for  how  long  the 
subject  remained  at  any  particular  location  in  the  VE.  This  allows  us  to  ask  such  questions  as:  Did  the  subject  spend 
a  great  deal  of  time  in  the  culvert  tunnel,  for  example,  indicating  perhaps  that  the  participant  was  taking  greater  care 
in  their  observations?  What  was  the  length  and  shape  of  the  path  they  traversed?  How  much  time  was  spent  in  the 
protected  culvert  compared  to  the  time  spent  in  open  space  outside  the  tunnel?  With  custom  queries  to  Phloem,  we 
can  look  into  the  data  and  ask  these  types  of  questions. 

As  well.  Phloem  can  facilitate  more  complex  custom  queries,  such  as  what  was  in  the  subject’s  field  of  view 
at  any  time  throughout  the  experience.  For  example,  we  may  want  to  know  if  a  participant  observed  a  particular 
object  of  interest,  such  as  a  roof  guard,  so  that  we  can  compare  this  sighting  to  whether  or  not  the  guard  was 
reported  in  the  Immediate  and  Delayed  Memory  assessment.  As  not  every  participant  has  the  same  experience  in  the 
outside  epoch  of  DarkCon  this  is  especially  important.  We  do  not  want  to  score  as  wrong  a  missed  report  of  an 
object  that  was  not  even  seen.  For  example  a  Phloem  query  can  account  for  how  long  the  sentry  on  the  roof  (or  any 
object)  was  within  the  field  of  view  (FOV)  of  the  participant.  We  can  assume  that  if  the  object  was  in  that  FOV  for 
a  certain  number  of  seconds,  the  chances  that  the  participant  perceived  the  object  are  very  good  (Although  “seeing” 
and  “perceiving”  cannot  ultimately  be  deemed  as  equivalent!).  Having  this  capability  allowed  us  to  correlate  specific 
behavioral  activity  with  the  physiological  data  being  captured  throughout,  as  well  as  with  participant  data  from  the 
self  report  questionnaires.  Phloem  allowed  us  to  account  for  individual  differences  in  the  free  will  experience  that 
might  not  have  been  possible  otherwise  and  it  is  possible  to  customize  any  queries  that  make  sense  for  any  particular 
study. 

4.2  Detailing  a  participant’s  physiological  response 

Phloem  allows  for  a  detailed  analysis  of  an  individual’s  unique  experience  in  a  VE.  To  illustrate  this,  we  will 
present  one  subject’s  experience  via  his  recorded  skin  conductance  response,  as  displayed  in  Figure  3. 
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Figure  3.  Annotated  Skin  Conductance  results  from  one  DarkCon  subject  obtained  via  a  replaying  of  the  experience 

with  Phloem 

With  SCR,  both  the  onset  of  a  stimulus-related  response  (i.e.  the  SCR  latency),  and  the  rise  time  (between  onset  and 
peak)  are  typically  between  1  and  3  seconds.  For  this  reason  a  time  scale  of  20s  with  a  minor  of  3  s  is  displayed  to 
help  read  the  annotated  SCR  graphs  corresponding  to  the  time  spent  inside  the  culvert  (154.8s)  and  outside  the 
culvert  (266.4s)  before  this  particular  subject  was  detected  by  the  roof  guard/sentry.  The  unit  for  the  SCR  is 
lOmV/pS,  which  is  the  gain  used  for  this  subject.  On  the  vertical  axis  1  unit  =  O.lpS. 

The  first  large  SCR  signal  displays  an  anticipatory  response,  i.e.  the  subject  had  just  started  out  in  the  dark 
tunnel,  and  was  looking  ahead.  Fie  sidestepped  and  walked  along  the  wall  towards  the  red  glow  of  an  alcove 
containing  a  pump,  which  was  heard  but  not  seen.  Just  as  the  participant  reached  the  alcove,  he  turned  his  gaze  to 
his  right  and  down,  noticing  debris  (in  this  case,  a  water  carrier  labeled  Voda).  After  a  turn  he  finally  made  a  visual 
contact  with  the  pump,  then  looking  back  over  to  his  left  and  down  he  noticed  a  baby  doll  in  the  mud.  Note  that 
when  the  participant  looked  at  details  (Voda  and  the  debris)  strong  SCRs  were  elicited.  The  participant  then  moved 
on  towards  overhanging  pipes  and  looked  more  closely  at  the  pipes  and  bats  hanging  from  them.  A  response  from 
the  bats  flying  away  is  observed  (which  we  designed  as  a  startle  stimulus).  As  he  continued  down  the  side  tunnel 
another  response  is  observed.  The  participant  saw  and  heard  stones  rolling  down  a  small  dirt  mound  while  hearing  a 
distant  (truck)  rumble,  which  produced  two  clear  SCRs. 

At  the  culvert  exit,  but  before  leaving  its  “safe”  confines,  the  participant  looked  outside  (left  and  right), 
with  an  observable  small  cluster  of  SCRs  and  simultaneously  had  a  pronounced  drop  in  HR  that  is  not  shown  on  this 
chart.  This  HR  deceleration  might  reflect  an  attentional  orientation  of  the  participant  to  the  novel  environment  of 
the  outside  world.  At  the  exit  of  the  culvert,  a  loud  truck  was  first  heard,  and  then  seen,  with  the  subject’s  gaze 


traveling  upwards  to  an  overhanging  bridge,  bringing  the  large  tmek  into  visual  eontaet  -  both  stimuli  produeing 
small  responses.  It  is  interesting  to  observe  the  responses  when  the  main  building,  the  wreeked  ear,  and  the 
damaged  building  are  seen.  Of  great  interest  to  our  study  is  the  subjeet’s  response  when  he  first  saw  the  lookout 
guard  on  the  rooftop.  It  seems  that  there  was  a  slight  eonfusion  in  deeiding  where  to  go  next  and  the  greatest 
response  appeared  when  the  subjeet  looked  baek  at  the  eulvert  entranee.  Did  he  feel  exposed  now  to  more  imminent 
danger?  Was  he  searehing  for  a  plaee  to  hide?  A  little  later,  this  subjeet  saw  the  roof  guard  again,  and  reaeted  by 
mnning  for  eover  behind  a  wreeked  ear  (plaeed  there  for  exaetly  this  purpose).  At  that  point  he  was  deteeted,  and 
the  end  of  the  seenario  was  played.  In  this  ease  the  seenario  was  ended  with  a  dog  attaek.  Overall,  this  subjeet’s 
SCR  indieates  responses  at  many  of  our  purposely  plaeed  emotional  hotspots. 

Both  HR  and  SCR  data  were  derived  over  the  three  time  epoehs  defined  earlier  and  reported  in  terms  of 
mean,  median,  and  standard  deviation  values.  In  the  ease  of  the  SCR,  averaging  the  SCR  data  points  over  the  epoeh 
of  ehoiee  derived  a  erude  estimate  of  the  SCR  mean  level.  Figure  3  illustrates  how  a  replay  of  the  VR  experiment 
helps  produee  an  overall  view  of  the  reaetions,  whieh  shows  a  wealth  of  event  related  response.  We  also  plan  to 
inelude  the  quantitative  information  eontained  in  the  SCR  that  is  not  yet  shown  here.  In  this  regard,  a  more  detailed 
analysis  of  the  SCR  is  eurrently  underway  to  seore  the  amplitude  and  lateney  for  eaeh  non-speeifie  response  using  a 
parameterization  software  paekage  (EDR  PARA,  Version  3.73)  by  German  researeher  Florian  Sehaefer.  This  will 
allow  us  to  report  the  sum  of  amplitude  (for  instanee,  exeeeding  a  0.15  pS  level)  and  the  ffequeney  of  non-speeifie 
responses  for  all  three  epoehs.  Note  that  onee  edited,  this  list  of  non-speeifie  response  parameters  also  permits  us  to 
look  at  singular  events  with  a  smaller  time  epoeh,  sueh  as  when  the  user  disturbs  a  ftoek  of  bats  that  abruptly  fly 
away  (i.e.  a  startle  stimulus). 

A  eomprehensive  statistieal  analysis  of  derived  data,  is  eurrently  in  proeess.  This  will  help  us  to  further 
determine  the  relationship  between  subjeet  eharaeteristies,  physiologieal  arousal,  behavior  and  the  memory  data  of 
the  partieipants  in  this  initial  study  and  is  expeeted  to  be  eompleted  in  June  2005. 

5  Conclusions  and  Future  Work 

Just  as  in  the  naturalistie  exploration  of  real  world  environments,  humans  who  experienee  free  will  exploration  of 
virtual  environments  demonstrate  a  wide  range  of  behaviors  and  responses.  This  is  to  be  expeeted.  It  is  preeisely 
these  individual  manifestations  of  behaviors  and  responses  within  a  virtual  environment  that  is  of  most  interest  to 
our  researeh.  Although  this  poses  eertain  methodologieal  ehallenges  that  go  beyond  what  is  typieally  found  in 
traditional  stimulus/response  studies,  it  is  exaetly  this  ageney  that  ean  be  one  of  the  assets  that  VR  offers  human 
researeh.  The  information  that  ean  be  extraeted  and  visualized  via  Phloem  is  invaluable  in  this  regard  and  may  offer 
a  more  preeise  approaeh  to  quantifying  naturalistie  behaviour  in  a  VE.  We  eventually  expeet  that  with  a  large 
enough  subjeet  population,  to  be  able  to  extraet  and  quantify  typieal  patterns  of  behavior  within  sueh  free  ageney 
worlds.  In  addition  to  expanding  the  library  of  eustom  queries  generated  over  time  with  this  applieation,  we  plan  to 
enhanee  Phloem  by  providing  the  evaluator  or  analyst  additional  tools  to  build  and  graph  queries  on  the  fly  during 
real  time  VE  interaetion  from  the  Phloem  interfaee.  This  will  allow  for  exploration  of  partieipants’  data  in  a  highly 
interaetive,  immediate  and  usable  fashion.  We  believe  that  eontinued  work  in  this  area  will  result  in  even  more 
robust  tools  to  effeetively  manage  and  analyse  the  eomplex  streams  of  data  that  are  possible  when  researeh  questions 
dietate  a  free  will  interaetion  format  within  VR  studies. 


Figure  4.  Seenes  from  the  Virtual  Iraq  exposure  therapy  applieation  for  eombat-related  PTSD 


For  example.  Phloem  will  be  integrated  into  an  ongoing  elinieal  applieation  projeet  that  is  targeting  the 
treatment  of  Post  Traumatie  Stress  Disorder  in  returning  Iraq  and  Afghanistan  military  personnel  (Rizzo  et  ah, 
2005).  In  this  applieation,  designed  to  deliver  graduated  exposure  therapy  within  the  eontext  of  VR  seenarios  that 
resemble  eombat-related  settings  (see  Figure  4),  elinieians  have  the  eapaeity  to  deliver  real  time  trigger  stimuli  while 
monitoring  the  ongoing  physiologieal  status  of  the  elient  on  the  same  interfaee.  This  integrated  interfaee  and  data 


capture  approach  is  essential  for  giving  the  clinician  the  capacity  to  modulate  client  anxiety  responses,  as  is  essential 
to  promote  the  therapeutic  habituation  that  is  desired  within  an  exposure  therapy  format.  Since  this  PTSD 
application  is  necessarily  a  free  will  exploratory  environment,  Phloem  will  be  an  essential  tool  for  developing  a 
“fine-grain”  analysis  of  the  psychophysiological  factors  that  underlie  this  disorder  in  order  to  design  more  effective 
treatment  protocols.  As  we  begin  to  integrate  Phloem  into  other  such  applications  and  share  the  tool  with  other 
researchers,  we  expect  its  functionality  to  evolve  and  more  comprehensively  address  such  free  will  VR  research 
methodology  challenges.  The  SEE  Project  is  at  the  beginning  of  this  long  term  research  program  and  we  anticipate 
that  the  Phloem  tool  will  be  of  benefit  to  the  many  in  the  community  of  VR  researchers  and  clinicians. 
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